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ABSTRACT 

The determination of final success of educational 
research and development projects is an important problem for those 
concerned with project management and evaluation. The purpose of this 
study vas twofold: (a) to determine if schedule^ cost# 
quality/performance, follow-on work^ spin-off benefits^ and 
customer/client satisfaction were given a different order of 
importance by different persons associated with education projects 
and what the order of importance was; and (b) to determine if a 
composite/criterion of educational project success employing the same 
criteria could be established. The data were obtained through 
mailings to project managers^ parent organization representatives^ 
and government project monitors of 218 terminated projects funded by 
the U.S. Office of Education. Sixty-five percent responded to the 
mailing by completing the data collection instruments. It was found 
that different persons associated with educational projects viewed 
the importance of the criteria of project success in much the same 
way, in the following order of importaace: (a) quality/performance; 
(b) qustomer/client satisfaction; (c) spin-off benefits; (d) 
follow-on work; (e) cost; and (f) schedule. (A section citing 
implications for project management and evaluation, suggestions for 
future research, and an appendix of data collection tools is 
includearj (JS) 
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1. Introduclioii 



Tlie determination of the final success of educatiomil rescarcli and 
development projects is an important problem for those concerned with 
project management and evaluation. Cne immediate problem encountered in 
determining project success is defining what is meant by "success." B>' 
what criteria is success measured? Does a successful project mean only 
tliat performance objectives have been achieved and/or that schedules and 
budgets were met? Are these three or other criteria more important? 
'Iliis paper liighlights some of the major results of a study concoraed with 
this problem. 

A. Criteria of Project Success 

Local school districts, state department.*^ of education, and the U. S. 
Office of Education have employed a wide variety of criteria in evaluating 
educational projects. Examples of these criteria are: 

1. inproved reading scores 

2. increased inter-ethnic group contact and luulcrstanding 

3. Improved physical, social, and emotiomil development of 
disadvantaged ptq3ils 

4. increased parent involvement 
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Research studies concerned with project success in cducntion have re^ 
suited in the identification of many criteria.^ lixamples of these are: 

1. constructive change 

2. meeting objectives 

3. innovative and creative 

4. serving a sizeable number 

5. meeting area needs 

6. development of skills 

7. acceptance by others 

8. acceptable level of output 

9. employment criteria met 
10. within Imdi^ct 

Although numerous criteria for detemininu project success have hccn 
identified many are vaguely defined if defined at all. Others are close- 
ly tied to funding agenc>^ requirements regarding hoiv the project is to be 
conducted. Many of these, such as -serving a sizeable number" and ''employ^ 
ment criteria met- really don't appear to be measures of project success. 
Rather, they appear ,z be indicators of whether or not the funds were spent 
according to funding agency regulations. Are all of these criteria of 
equal importance? 



Schalock^^^r.^f '""'"^if n^'^^'^'f ^^"^^^^ ^he work: 11. Del 

Bmi^ton 1j ^" Fdnr..t^ onal Research. Develop ment. 

JJittusion and Evaluation- -voTnm P t <^nm^r D;. p^^^ fHonmniith Oiluoii ' 

iittihT ' °^ s^ateVst rrgM^^^^ 
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vSucccss as a Dependent Variable 

Once it is known what constitutes success, researdv can be directed 
toward discovering those variables in the management of projects critical 
to determining a successful project. The determination of what constitutes 
a successful project is also a necessary initial step in the evaluation of 
projects. If success is to serve as a dependent variable in project manage- 
ment research and evaluation, then a valid and reliable measure of it is 
necessary. 

2 

One of the objectives in a study reported by Marquis and Straight in 
1965 was to determine what was considered success in the performance of 
research and development projects. The laboratory manager, project manager, 
government technical monitor, and government contract administrator associat- 
ed with each of thirty-seven projects were asked to identify their criteria 
of project success and to rank them in order of inportance. Technical per- 
formance was ranked first by the majority of respondents followed by meeting 
schedule, meeting target cost, customer satisfaction, profit, follow-on 
business, company prestige, develop technical capability, and commercial 
applications. Marquis and Straight chose to use cost overrun, schedule 
overrun, and teclmical performance as indicators of project success in ex- 
ploring relationships between project organizational variables and project 
success. 



in Prn?^^? 1 ;r ^^^^^ ^' Straight, Jr., "Orgainzational i-actors 
xn Project Performance," Research Program Effectiveness- -Pr oceeding's of the 
Conference Sponsored by tHT Office of NavaT Research Av;,^hi'natnn ^Qf.'^^ 
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In 1973, Baker and others reported the results of a study designed to 
specify the relationships ainong situational, organizational, and project 
management process variables as they relate to project success. Data were 
gathered by questionnaire from 646 persons involved in project management 
wi thing business, industry, and government. The factor analysis of the ques- 
tionnaire responses resulted in the identification of a factor that was sub- 
sequently used as a measure of project success. The factor was comprised of 
questionnaire items relating to client, parent organization, project team, 
and user satisfaction with the outcome of the project; the technical adequacy 
of tihe project end result; and a statement that all things considered, the 
project was a success. 

The studies of Marquis and Baker illustrate the inportance of specify- 
mg project success in the conduct of project management research and 
evaluation. Research studies such as tlies^d^ve identified other numerous 
criteria subsequently enployed in the measux*ement or determination of pro- 
ject success. 



C. Framework for the Study of Project Management 

In 1970, Cook conducted a research study resulting in a conceptual 
framework (illustrated in Figure 1) to he used for the study of project 
manageTOnt. The purpose of the effort was to integrate the results of 



— s 

Bruce N. Baker, Dalmar Fisher, and David C. Murphy, "Factors Affecting 
Success of Project Management,** Proceedings of the Fifth Annual Seminar - 
Symposiura of the Project Management Institute, [Toronto, Canada, 1973). 

Desmond L» Cook, **A Conceptual Framework for the Study of Project 
Managanent,** Proceedings of the TTiird Annual Seminar -Symposium of the Project 
Management Institute (Houston, Texas, 1971), 




past and current studies and to direct further research on project niaiuigo- 
nient. IVoject success represerts a mjor component of the concent oli/.ntion. 
It includes the potential criteria defined below. 

1. Schedule refers to meeting project milestone deadlines and 
the scheduled project termination date as established in the project con- 
tract. Success on this d.unonsion meajis that the project was completed on 
or before the scheduled termination date imd that all subsections or com- 
ponents of the project were completed as scheduled. 

2. Cost refers to meeting the total dollar expenditure predicted 
for the entire life of the project. Success on this dimension means tliat 
the actual dollar cost of the project was less than or coincided with the 
predicted cost, 

3. (jiutlity/l'eri'oniuuice refers to the achicvomoiit ol" ihc };";'1^> •''"kI 
objectives of the project. Success on this iILnonsion metui.s tluit the perlor- 
inancc stajidards specified by tJie project objectives were met or exceeded. 

4. Customer/Client Satisfaction refers to the degree to which 
the funding agency or user of the end product of the project is satisifetl. 
This criterion is not necessarily concerned with the quality of the project 
as indicated by whether or not the original objectives and performance stan- 
ards were met. It is concerned only witli the satisfaction of the funding 
agenry or customer. 

5. Spin-off refers to indirect benefits received by the parent 
institution, its faculty, and the project staff. Success on this dimension 
means, for example, that the project and/or the parent institution's faculty 



BEST COPX AVAILABLE 



o 

lU 



lU 



LU 

O 

> 

2: 

LU 



Si 



•IS 

o 

*^ I 
la- Of 



«) 

0) 

u 

CO 



o 
o 



g 












U 












g c/> 


a 




t 


^ 4J 


c 








*a 


O 




• 


• 



I 
I 



I 
I 
I 
I 



I 

I C 

ca o 
c *~ 

u 

u o ^ 
•■n Q) 

o u jm: 

I- Q) «j 
CL cnx 





















c 








0) 






• * 


6 






in 




I 




c 








u 




Id 




0) 




o 




o 




u 


:> 


c 




CL 




o 


« 


• 




o 









c 
o 



O 0) 
CL 



u 

3 
U 
0) 

to 



c 
u 




Q) «J —I 



-7- 



developecl skills as a result of working with the project which can be ap- 
plied to other areas of the institution's operations. Another important 

« 

spin-off benefit is the recognition an institution often gains from a highly 
successful project. 

6. Follow-on IVork refers to additional contracts or follow-on 
work obtained as a result of the success of the original project and con- 
sequent enJiancement of the reputation of the researcher. Tlus criterion is 
not, however, concerned with additional work obtained as a logical or tech- 
nical extension of the original project. 

In addition to identifying potential criteria for assessing project 
success. Cook's conceptual fraiiiework suggests that different persons may have 
different viewpoints on project success, and that these would be reflected in 
their ranking of the potential success criteria. For exan^le, project mana- 
gers may be most concerned with time, cost, and performance, whereas, parent 
organization representatives (administrators, deans, department chairman) 
may be quite concerned about spin-off benefits, follow-on work, and cus- 
tomer or funding agency satisfaction. 

n. Purpose 

The purpose of this study was to explore two basic hypotheses or assump- 
tions suggested by Cook's conceptual framework in regard to the criteria of 
project success. Hie major objectives of the study were: 

1. To determine if scliedule, cost, quality /perfbrmcu>ce, follow- 
on work, spin-off, and customer/client satisfaction are given 

ERIC 



a different order of rinportance by different persons ;ibso-^ 
ciated with educational projects and what the order (s) of 
importance is (are). 
2. To determine if a composite criterion of educational proj- 
ect success employing the criteria of schedule, cost, 
quality/performance, follow-on work, spin-off, and customer/ 
client satisfaction could be established, 

III. Procedures 

A sajiple of educational projects was first selected and then the 
personnel associated with the conduct and nmitoring of the projccis were 
identified as the source of data for the study. The instruments ased in 
the collection of data, the sanq)le and data collection procedures, and the 
data analysis procedures are briefly described below. 

A. Instruments 

A Criteria Ranking Form using a paired- comparison format was developed 
to obtain rankings of the relative imj^ortance of the potential criteria of 
project success. A Project Rating Form using a seven-point, equal -interval 
scale was developed to obtain ratings of actual projects on the potential 
criteria and overall success. Copies of these instruments appear in the 
Appendix. 

Some of the hypotheses of the study (not reported here) required that 
information be collected regarding the parent organization hoasing the pro- 
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ject, the previous experience of the project principal invest ii'.ntor, rhe 
siqiervision given the project by others, ;in(l the fcunillarlty of the respon- 
dent with the project being rated, 

B, Sample and Data Collection 

^^'.0 data for the study were obtained from the pioject manajjer, parent' 
organization representative, and government project monitor associated with 
each project in a sample of 218 terminated projects funded in one of the 
several regions under the Regional l^esearch Program of the National Center 
for Educational Research and Development, United States Office of Hducation. 
The projects were all operated and terminal ted during' the period from September, 
1967 to January, 1972; had a maximm government support of $10,000, and were 
usually conducted or managed by a single "principal investigator." 

Tlie data v>;ere collected in 1972 by means of six initial imiilings and a 
follow-up mailing to all non- respondents for whom correct mailing addressee 
could be assujiied. Sixty-five percent of the project principal investigator-, 
and parent organization representatives responded to the mailing by completing 
the data collection instruments. Ninety-five percent of t'^e Criteria Rajiking 
Forms received were usable, and 93 percent of the Project Uting Forms were 
usable. The one government monitor associated with the entire sajnple of 
projects responded by completing the Criteria Ranking Form and numerous Project 
luting Forms* 
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C* Data Analysis 

In order to obtain a ranking of the six criteria of project success, 
the paired -comparison data were run on the PCMP computer propram of the 
Psychology Department of The Ohio State University* This program follows 
the assunptions of the Thurstone Case III model and computes a scale value 
for each criterion* 

In order to measure the agreement within groups of respondents, a 
coefficient of agreement, u, was computed*^ The chi-square test of sig- 
nificance of u gives one a level of confidence in stating that the agree- 
ment of a group in ranking something is not the result of chance, but 
does, in fact, indicate some agreement among the members of the grouj)* 

In order to measure the agreanent between different groups of re- 
spondents in their rankings of the relative importance of the criteria, 
an analysis of variance procedure was used in estimating the reliability 
of scale values given the criteria by the groups This represents the 
degree of agreement of the groups or judges in the scale values they have 
given the criteria. 

In order to obtain a conposite criterion of project success, the rat- 
ings of projects on the six criteria plus overall success made by the re- 



"l^urice G. Kendall, Rank Correlation Methods (London: Charles 
Griffin and Coirpany Limited, 1948J, page 126* 

Robert L* Ebel, "Estimation of the Reliability of Ratings," 
Psychometrika , 16, 4 (December, 1951): pages 407-424. 
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si)ondonts were rm on the h-heriy Test Selection Program of the Psychology 
Department of "me Ohio State University. The program is a regression 
analysis program which selects, in order of decreasing importance, the 
variables contributing significant variance to the prediction of the 
criterion variable (overall success) . 



rv. Results 



It was found that different groups of persons associated with edu- 
cational projects viewed the relative importance of the six possible 
criteria of project success much the s:unc. 'ITierc may be some diffcrciiccs 
between individuals, but as a groq) project principal investigators and 
parent organization representatives view the criteria in the following or- 
der of inportance. 

1 . Quality/Performance 

2. Ciistomer/Client Satisfaction 

3. Spin-off Benefits 

4. Follow- on Work 

5. Cost 

6 . Schedule 

The scale values for the ranking of the criteria by the two groups 
are given in Table 1. Tlie coefficient of agreement, u, within the groups 
was 0.373 for the 136 principal investigators and 0.415 for the 72 parent 
organization representatives. The significance of the chi-square value 
(.001 level) in both cases indicated a level of agreement (1.00 is complete 
agreement) among the group members not attributable to random choices. 
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The agreement in rankings between the groups of project principal 
investigators and parent organization representatives is illustrated in 
Figure 1. A coefficient of agreement or reliability of 0.96 was obtained 
between the two groups. 

The conposite criterion of overall project success generated for 
project principal investigators and parent organization representatives 
is summarized in Tables 2 and 3. Note that the shrunken multiple R changed 
very little after two or three variables were included in the prediction 
equation. These variables were quality/performance, customer/client sat- 
isfaction, and spin-off benefits. The maximum shrunken multiple R ob- 
tained in each case was a little over 0,700 which indicates that the three 
criteria are accounting for approximately SO percent of the variance in 
the criterion variable of overall project success. 
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TABLi: 1 

SCMJ-: VAIiJlIS POR RANKING OF PROJECT SlICCl-SS CRI'ITJUA 
MADE BY PRINCIPAL INVESTIGATORS AND PARIOT ORGANIZATION RliPRESl-MATlVES 
ASSOCIATEJ) WnH A SAMPLE OF EDUCATIONAL PRQJIiCTS 



Criteria 


Principal 
Investigator 
(N=136) 


Parent 
Organization 
Representative 
(N=72) 


Quality 


2.97 


2.81 


Customer Satisfaction 


1.87 


1.61 


Spin-off 


1.61 


0.99 


Follow -on Work 


1.25 


0.89 


Cost 


0.00 


0.32 


Schedule 


0.03 


0.00 
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Judge Reliability 0.9559 



Figure 2, Comparison of scale values given to criteria of project suc- 
cess by principal investigators and parent organization representatives 
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V. TniiUications for Project Management and nvaluation 

'ITie most important criterion in determining the success of a project 

appears to focus upon quality /performance, llais finding is supported by 
7 

the Baker study. To have a successful project, the project manager should 
concentrate on the achievement of the goal and objectives of the project. 
Management decisions are said to usually involve trade-offs between time, 
cost, and performance. Since time or scliedule and cost were ranked the 
lowest among the six criteria of project success considered, the project 
manager should try to maximize quality/performance when faced with dcciiiions 
involving trade-offs between time, cost, and performance. 

The parallel implication for project monitors, contract administra- 
tors, and evaluators is not to attach high importance to meeting project 
schedule and cost constraints. Although completing a project on schedule 
ajul within budget may appear to be important, achieving the goal and objec- 
tives of a project is far more important. Consequently, time and cost 
factors should not be unnecessarily constrained to the detriment of pro- 
ject performance. 

Although much writing in project management literature stresses the 
importance of time, cost, and performance factors to the project manager, 
there are other very important factors that deserve consideration. Among 
these are customer/client satisfaction, spin-off benefits, and follow-on 

7 

Bruce N. Baker, Dalmar Fisher, and David C. Murphy, ''Factors 
Affecting Success of Project Management,'* page 4. 
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work, 'lliese factors may appear to be external to the project and of 
secondiiry importance, but the findings indicate that they are of substan- 
tial iin{^ortance in judging tlie success of a project. 'ITic project manziger, 
needs to concern himself with the likely effects of his decisions upon 
the satisfaction of the funding agency cind ultinuite client or user with 
the end product of the project, lie should also be concerned with the 
indirect benefits the project may Ime for the pa: ant organization, its 
staff, and the project staff. Finally, he should be concerned with the 
implications that project success has on obtaining other additional or 
follow-on projects for himself and/or his parent organization following 
the termination of the project. 

The parallel inplication for project monitors and evaluators is to 
give consideration to these factors when determining the success of a 
proj ect. 

In summary, the major implication or recommendation for the educa- 
tional project manager is to achieve the project goal and objectives even 
if one has to overrun the schedule and budget. Relative to other cri- 
teria, time and cost are not very important. And, while one is striving 
to achieve the objectives of the project, try to ensure that the customer 
or user will, in the end, be satisfied. Also try to maximize spin-off 
benefits such as the development of skills in the project staff and new 
capabilities in the parent organization. Finally, don't overlook the im- 
portance of trying to obtain future projects for the parent organization. 
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VI > Suggestions for Further Research 



Based i^on tlie limitations and findings and conclusions of this 
study, the following recommendations for future research related to de- 
termining the success of educational projects are made. 

1, Quality/performance is considered to be by far the most 
important criterion of educational project success. Meas- 
urement on the criterion is most often achieved by obtain- 
ing the opinion or judgment of knowledgeable individuals, 
Coasequently, the reliability and validity of the evalua- 
tion of projects on this criterion is often open to ques- 
tion. Research directed toward in^^roving the reliability 
and validity of measurement on this important criterion is 
needed, 

2, This study dealt with projects of a limited size or 
scale from a limit geographic region that were nearly all 
conducted in a college or university setting. Research 
is needed to determine if the results hold for larger, 
more complex projects conducted in different educational 
institutions across the country. For example, future 
studies might include large, state or federally fionded 
projects in large, city public school districts in t-heir 
sample, 

3, The variability of the projects included in the study on 
the criterion of cost was limited by the fact that tlie 
contracts were all of the fixed price type. It iz quite 
possible that the inqx)rtance of the criterion of cost may 
vary with the type of contract issued. Future studies 
should include projects operated urJer a variety of con- 
tract types, 

4, Studies directed at discovering which variables in the 
management of educational projects are critical in de- 
termining a successful project, should use a composite 
criterion as the dependent variable which includes the 
factors of quality/performance, spin-off benefits, and 
customer/client satisfaction. 
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Success Criteri<i Ranking 

Listed below in palr$ are the criteria of education^) project buccoss thot were 
defined on the previous page . Select the criterion in each pair that you feel 
is the more important of the two in judging the success of a project by placing 
an "x" before the more important criterion in each pair After you have finished, 
turn the page and continue. Be sure that all items are marked. 



1. customer/client satisfaction 

^cost 

2. ^qual I ty/pcrformance 

^schedule 

3. ^cost 

^spin-off 

schedule 

follow-on work 

5. ^spin-off 

<qual i ty/performance 

6. customer/client satisfaction 

schedule 



10. 
IK 
12. 
13. 



jqual ity/performanc^ 
_custoi{.er/cl lent satisfaction 

_spln-of f 

_fol low-oa work 

_cos t 

jqual ity/performance 

_customer/cnent satisfaction 
_fol low-on work 

_schedul e 
_spln-of f 

^fol low-on work 



7. follow-on work 

^Qual i tv/norformance 

8. schedule 

cost 



15. ^spin-off 

^customer/client satisfaction 



Check the bok at the right if you would like to receive a copy of the results 
of this study of educational project success criteria. 
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